Apparatus and methods for monitoring vehicles

ABSTRACT

An apparatus and method for monitoring the status and health of a fleet of vehicles operating in a common space. A centralized monitoring operator receives status information and has the capability to independently interact with each vehicle in the fleet.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Application No. 17/034,365,filed Sep. 28, 2020. Now, U.S. Pat. No. XXXXX which claims priorityunder 35 U.S.C. § 119(e) to U.S. Provisional Pat. Application No.62/907,933, filed Sep. 30, 2019, which are expressly incorporated byreference herein.

TECHNICAL FIELD

The present disclosure is related to the operation and monitoring of afleet of autonomous vehicles. The disclosure relates to centralizedmonitoring of an active fleet of autonomous vehicle and methods forremotely diagnosing and resolving errors in operations of specificautonomous vehicles through augmented reality.

BACKGROUND

The use of robotic or autonomous vehicles in manufacturing and materialhandling applications is expanding rapidly as technology advances. Theability to adapt software to autonomous vehicles to improve decisionmaking and perform more and more complex tasks is a driver ofproductivity. For example, having autonomous vehicles in a fleetcooperate and avoid dangerous interactions allows more autonomousvehicles to be used in a single workspace.

The ability for fleet members to interact and complete more complextasks also presents additional opportunities for autonomous vehicles toencounter error conditions that they are neither programmed to recognizeor resolve. Unexpected conditions, such as an unexpected impediment in aworkspace or a malfunction of a particular subsystem of the autonomousvehicle can present problems that magnify within a fleet environment,such as for example, if a malfunctioning autonomous vehicle blocks theworkflow of other autonomous vehicles in the fleet.

Failures of autonomous vehicle subsystems or error conditions cancripple a fleet operation in short order if not resolved. In many cases,resolution requires a human interaction that requires the human to enterthe workspace and resolve the issue. The introduction of the human intothe autonomous vehicle workspace presents new obstacles for the fleetmembers as well as potentially exposing the human to injury by one ofthe autonomous vehicles. This is confounded by the issue that manyautonomous vehicles do not have readily available user controls tooperate the autonomous vehicle and, even if present, the autonomousvehicle is not likely to be adapted for an operator to manually operatethe autonomous vehicle.

Still further, the need to have humans on “stand-by” reduces theproductivity gains provided by a fleet of autonomous vehicles. Whenoperating properly, autonomous vehicle fleets need little to nointervention in the normal activities. For example, autonomous vehiclesmay be programmed to return to charging/fueling stations when needed andmay take themselves out of service when preventative maintenance isrequired. Thus, there is no need for humans to be present when thefleet, or any particular autonomous vehicle, is operating correctly.There is a need to have humans available in the vicinity of theautonomous vehicles if error conditions arise.

Improved performance of autonomous vehicles and the reduction of theneed for nearby humans to resolve errors and system malfunctions inautonomous vehicles would provide additional productivity and costreductions, as well as reducing the potential for injury to humans.Still further, due to the limited number of issues that occur in aparticular workspace, reducing or eliminating the need for humanphysical interaction presents the potential for additional productivitygains.

SUMMARY

The present disclosure includes one or more of the features recited inthe appended claims and/or the following features which, alone or in anycombination, may comprise patentable subject matter.

According to an aspect of the present disclosure, an autonomous vehicletracking system comprises at least one autonomous vehicle and acentralized monitoring station. The at least one autonomous vehicleincludes a control system. The control system includes a drive system, aplurality of sensors, a camera, and a controller. The drive systemprovides power for the autonomous vehicle and is operable to move theautonomous vehicle and to operate accessories of the autonomous vehicle.The sensors provide signals indicative of real-time informationregarding the location and operation of the autonomous vehicle. Thecamera system provides signals representative of images of theenvironment surrounding the autonomous vehicle. The controller includesa processor and a memory device. The memory device includes instructionsthat, when executed by the processor, cause the processor to receivesignals from the sensors and the camera system, aggregate the signals tocreate an array of data that is a composite of the sensor signals andthe camera signals that represents a time-sequenced composite image filethat superimposes data derived from the sensor signals onto the imagesfrom the camera system. The centralized monitoring station includes acomputer, a user input device, and a display device. The computer is incommunication with the controller of the at least one autonomousvehicle. The computer includes a processor and a memory device. The userinput device is in communication with the computer. The display deviceis in communication with the computer. The memory device of the computerincludes instructions that, when executed by the processor, cause theprocessor to process inputs from the user input device to communicatewith the controller of the autonomous vehicle to prompt the controllertransmit portions of the time-sequenced composite image file under thecontrol of the user input device, the transmitted image file beingreceived by the computer and displayed by the display device.

In some embodiments, the time-sequenced composite image file istransmitted in real time and the portion transmitted varies based on theuser input to generate a field of view perceptible to a human.

In some embodiments, the display device comprises a head set and theuser input device is coupled to the headset such that movement of thehead set changes the portion of the image file transmitted responsive tomovement of the headset to change the field of view being displayed bythe head set.

In some embodiments, the user input device includes an input for varyingthe point in time which corresponds to the portion of the image filebeing transmitted, such that a user may choose to view the portion ofthe image file as it existed a different time from current real-time.

In some embodiments, the portion of the image file transmitted isresponsive to the position of the head set, such that the user may varythe field of view at the different time to view the surroundings of theautonomous vehicle at that point in time.

In some embodiments, the time sequence may be paused such that a usermay look around the autonomous vehicle at a single point in time bymoving the head set to change the field of view.

In some embodiments, the system includes a plurality of autonomousvehicles and the computer is in communication with the controller ofeach of the autonomous vehicles such that a user may select any of theplurality of autonomous vehicles and view the surroundings of theparticular autonomous vehicle using the user input device and the headset.

In some embodiments, the centralized monitoring station is operable toprovide mission tasks to each of the plurality of autonomous vehicles,the centralized monitoring system including a monitor that providescurrent status of each of the plurality of autonomous vehicles.

In some embodiments, the system includes a plurality of cameraspositioned in the working environment of the plurality of autonomousvehicles, the cameras providing a signal representative of images ofportions of the working environment of the plurality of vehicles, thecameras including memory to store an array of data that represents atime-sequenced image file for the field of view of the particularcamera.

In some embodiments, the computer is operable to transmit a signal toeach of the autonomous vehicles and cameras to simultaneously change thepoint of time that the time-sequenced images are presented such that anoperator may toggle between the views of each of the autonomous vehiclesand each of the cameras at a coordinated point in time, the portion ofthe image file being transmitted by each autonomous vehicle beingresponsive to the position of the head set of the user.

In some embodiments, the position of the head set is calibrated from aparticular datum in the environment of the plurality of autonomousvehicles.

In some embodiments, the position of the head set is calibrated from aneutral position relative to the particular camera or autonomousvehicle.

In some embodiments, each of the plurality of autonomous vehicles isoperable to transmit an alert condition to the centralized monitoringstation, the alert condition prompting the alert to be logged to theparticular real time of the alert, and wherein the computer is operableto mark the point in time such that a user may choose the time of thealert to view the images from the cameras or autonomous vehicles.

According to another aspect of the present disclosure, an autonomousvehicle comprises a control system including a drive system, a pluralityof sensors, a camera system, and a controller. The control systemincludes a drive system, a plurality of sensors, a camera, and acontroller. The drive system provides power for the autonomous vehicleand is operable to move the autonomous vehicle and to operateaccessories of the autonomous vehicle. The sensors provide signalsindicative of real-time information regarding the location and operationof the autonomous vehicle. The camera system provides signalsrepresentative of images of the environment surrounding the autonomousvehicle. The controller includes a processor and a memory device. Thememory device includes instructions that, when executed by theprocessor, cause the processor to receive signals from the sensors andthe camera system, aggregate the signals to create an array of data thatis a composite of the sensor signals and the camera signals thatrepresents a time-sequenced composite image file that superimposes dataderived from the sensor signals onto the images from the camera system.

In some embodiments, the time-sequenced composite image file istransmitted in real time and the portion transmitted varies based on auser input to generate a field of view perceptible to a human, the fieldof view changing based on the user input.

In some embodiments, the user input varies the point in time whichcorresponds to the portion of the image file being transmitted, suchthat a user may choose to view the portion of the image file as itexisted a different time from current real-time.

Additional features, which alone or in combination with any otherfeature(s), such as those listed above and/or those listed in theclaims, can comprise patentable subject matter and will become apparentto those skilled in the art upon consideration of the following detaileddescription of various embodiments exemplifying the best mode ofcarrying out the embodiments as presently perceived.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description particularly refers to the accompanying figuresin which:

FIG. 1 is diagrammatic floor plan of a material storage facilityincluding a autonomous vehicles used to move materials between storagelocations, a number of cameras positioned throughout the facility, and aremote monitoring station;

FIG. 2 is a block diagram of the components of an illustrativeautonomous vehicle of FIG. 1 ;

FIG. 3 is a flow chart showing an algorithm used by the autonomousvehicle of FIG. 1 to manage sensor data from various sensors of theautonomous vehicle;

FIG. 4 is a block diagram of the remote monitoring station of FIG. 1 ;

FIG. 5 is diagrammatic representation of an augmented reality interfacefor a user positioned in the remote monitoring station;

FIG. 6 is a perspective view of a portion of a working environment asviewed from the augmented reality interface, with the field of view ofthe augmented reality interface superimposed to show the limitations ofthe field of view of augmented reality interface at a particular momentin time;

FIG. 7 is a composite image of the field of view from an autonomousvehicle as viewed by the augmented reality interface including dataelements superimposed on the image to provide the user of the augmentedreality interface with additional information;

FIG. 8 is diagrammatic floor plan similar to FIG. 1 , FIG. 8 showing anobstruction positioned in the path of an autonomous vehicle;

FIG. 9 is a composite image similar to FIG. 7 showing an obstruction inthe path of the autonomous vehicle, the obstruction outside of the fieldof view of the augmented reality interface;

FIG. 10 is the composite image of FIG. 9 with the field of view adjustedto show the obstruction;

FIG. 11 is an example of a monitor screen of the remote monitoringstation in normal operating conditions;

FIG. 12 is an example of a monitor screen of the remote monitoringstation in an alert condition that has been identified due to thedetection of an impact;

FIG. 13 is diagram of the data flow for creating the composite image ofFIG. 7 ; and

FIG. 14 is a diagram of the communications between a particularautonomous vehicle and the remote monitoring station.

DETAILED DESCRIPTION

According to the present disclosure, a material storage facility 10includes a number of storage racks 12, 14, 16, 18 positioned in aworkspace 20 of the material storage facility 10. The material storagefacility 10 also includes a number of autonomous vehicles 22 thatoperate within the material storage facility 10 to process materials inthe material storage facility 10, including, in some embodiments, movingmaterials 24 from one storage location 26 to another storage location28. It should be understood that the autonomous vehicles 22 may alsomove materials from the particular material storage facility 10 toanother location outside of the material storage facility 10, such as toa manufacturing or distribution location outside of the material storagefacility 10. In the embodiment of FIG. 1 , the material storage facility10 is illustrative only and the equipment and methods disclosed hereinmay be applied to the use of autonomous vehicles 22 in variousfacilities and applications.

The material storage facility 10 further includes a remote monitoringstation (RMS) 30 positioned in the material storage facility 10. Anoperator 8 is located in the remote monitoring station 30 andresponsible for monitoring the fleet of autonomous vehicles 22 andresolving issues or error conditions. The positioning of the remotemonitoring station 30 in the material storage facility 10 is forillustrative purposes only. The remote monitoring station 30 may bepositioned outside of the material storage facility 10 at the samelocation, or may be positioned in a geographically distant location,such as a single remote monitoring station 30 facility that providesmonitoring for various material storage facilities 10 around the world.The material storage facility 10 also includes a number of fixed cameras32 positioned throughout the workspace 20 and positioned to provideviewing coverage of the workspace 20 in which the autonomous vehicles 22operate.

Referring now to FIG. 2 , the autonomous vehicle 22 includes acontroller 34 that is responsible for operation of the particularautonomous vehicle 22. The controller 34 includes a processor 36 andmemory 38. The processor 36 is in communication with the memory 38 andthe memory 38 includes instructions that, when executed by the processor36, cause the controller 34 to control the functionality of theautonomous vehicle 22. The processor 36 may be embodied as any type ofprocessor capable of performing the functions described herein. Forexample, the processor 36 may be embodied as a single or multi-coreprocessor(s), a single or multi-socket processor, a digital signalprocessor, a graphics processor, a microcontroller, or other processoror processing/controlling circuit. Similarly, the memory 38 may beembodied as any type of volatile or non-volatile memory or data storagecapable of performing the functions described herein. In operation, thememory 38 may store various data and software used during operation ofthe controller 34 such as operating systems, applications, programs,libraries, and drivers. The memory 38 is communicatively coupled to theprocessor 36 via the I/O subsystem 40, which may be embodied ascircuitry and/or components to facilitate input/output operations withthe processor 36, the memory 38, and other components of the controller34 or other components of the autonomous vehicle 22. For example, theI/O subsystem 40 may be embodied as, or otherwise include, memorycontroller hubs, input/output control hubs, firmware devices,communication links (i.e., point-to-point links, bus links, wires,cables, light guides, printed circuit board traces, etc.) and/or othercomponents and subsystems to facilitate the input/output operations. Insome embodiments, the I/O subsystem 40 may form a portion of asystem-on-a-chip (SoC) and be incorporated, along with the processor 36,the memory 38, and other components of the controller 34 on a singleintegrated circuit chip.

Autonomous vehicle 22 includes a number of sensors 150 providing data tothe controller 34. The sensors 150 include an array of cameras 42positioned in various locations on the autonomous vehicle 22 to providea three-hundred-sixty degree view of overlapping fields of view toprovide an remote operator 8 full access to view the surroundings of theautonomous vehicle 22. The cameras 42 are embodied as RGBD cameras,providing enhanced information controller 34 that will be used asdiscussed below. In some embodiments, the autonomous vehicle may alsoinclude an array of RGB cameras 142 that provide visual data without theenhanced depth of the RGBD cameras 42. The autonomous vehicle 22 alsoincludes one or more on-board inertial measurement units (IMUs) 44 toprovide input to the controller 34 regarding changes in acceleration andorientation of the autonomous vehicle 22 in three-space during use.Still further, the autonomous vehicle 22 also includes an array ofcontact sensors 46 positioned on the autonomous vehicle 22 so that anyphysical contact between the autonomous vehicle 22 and some other itemin the environment can be detected. Using the array of cameras 42,accelerometer(s) 44, and contact sensor array 46, the controller 34 isoperable to sense the environment around the autonomous vehicle 22 tocontrol the autonomous vehicle 22 as it completes its mission. Inaddition, the array of cameras 42, accelerometer(s) 44, and contactsensor array 46 allows the controller 34 to identify unexpectedconditions that may require a response by the autonomous vehicle 22 oran intervention by an operator 8 positioned at the remote monitoringstation 30. In some embodiments, the autonomous vehicle 22 may alsoinclude an array of LiDAR sensors 54 positioned about the autonomousvehicle 22 and operable to provide additional information to thecontroller 34 regarding the environment about the autonomous vehicle 22.Still further, in some embodiments, the autonomous vehicle 22 may alsoinclude an array of sonar sensors 56 positioned about the autonomousvehicle 22 and operable to provide additional information to thecontroller 34 regarding the environment about the autonomous vehicle 22.Additional sensors or sensor systems 150 of the autonomous vehicle 22may include encoders 144 for measuring movement of components of theautonomous vehicle 22 or radar sensors 140. Other sensors 152 may beused as required by a particular application.

Also shown in the block diagram of FIG. 2 , the autonomous vehicle 22includes a drive system 49 which is operated, under the control of thecontroller 34 to move and steer the autonomous vehicle 22. The drivesystem 49 includes a drive 50 controlled by a drive controller 48 thatcommunicates with controller 34 and provides the control signals for thedrive 50. It should be understood that the autonomous vehicle 22 couldhave other functionality, in addition to the drive 50, including amanipulation system 136 which may embodied as various implements such afork-lift mast system, articulators to move materials onto and off fromthe autonomous vehicle 22, or any of a number of other functions andsub-systems that adapt the autonomous vehicle 22 for specific purposes.The variation in functionality is limited only by the technicalcapabilities of a particular physical arrangement and the principlesdisclosed herein may be used with any fleet of autonomous vehicles 22that interact within a pre-defined workspace, such as workspace 20, forexample.

The autonomous vehicle 22 also includes communication circuitry 52 thatallows the autonomous vehicle 22 to interact with the remote monitoringstation 30 and to share data there between. The communications circuitry52 can take many forms and be similar to the input/output controller 40or be modified for a particular application. It is contemplated that thecommunications circuitry 52 would include a high-speed wirelesscommunications protocol to allow data to be shared between thecontroller 34 and the remote monitoring station 30 in real-time to allowthe autonomous vehicle 22 to operate at relatively high speeds in theworkspace 20.

As will be explained in further detail below, the array of cameras 42,accelerometer(s) 44, and contact sensor array 46, and optionally theLiDAR array 54 and/or sonar array 56 provide signals to the controller34 that are then processed and stored in memory 38 to provide a timesequenced data based on the inputs from the various sensors 42, 44, 46,54, and 56 that can be combined to provide a historical composite of thedata that can create a composite image and status that allows a user toreconstruct the environment of the autonomous vehicle 22 at an earlierpoint in time. This is useful for an operator 8 to diagnose a particularproblem experiences by an autonomous vehicle 22 by viewing thesurroundings of the autonomous vehicle 22 at an earlier point in time.The data can also be combined to create a composite image available tothe operator 8 at the remote monitoring station 30 in real-time so thatthe operator 8 may manually operate the autonomous vehicle 22 using thecombined data.

The autonomous vehicle 22 also includes an independent safety system 138with safety rated sensors that are operable to limit operation of theautonomous vehicle 22. The autonomous vehicle 22 includes a battery 88which powers the various components of the autonomous vehicle 22. Thebattery 88 interfaces with a battery relay 134 which is operable tointerrupt power to both the manipulation system 136 and drive system 49to prevent operation of those systems in critical safety situations. Ifthe safety system 138 detects an unsafe condition, the power to themanipulation system 136 and drive system 49 is cut until the unsafecondition is resolved.

The remote monitoring station 30 is shown diagrammatically in FIG. 4 andincludes a workstation 56 that includes a computer 152, a display 58, aQWERTY keyboard 60, and a mouse 62. The display 58 may include multiplediscrete display devices or may be segmented to provide views of statusscreens of multiple material storage facilities 10 simultaneously. Forexample, referring to FIG. 8 , a status screen of a single materialstorage facility 10 is shown. The display 58 may be configured tosimultaneously show each material storage facility 10 the operator 8 ismonitoring updated with real-time location and status indicators of thevarious autonomous vehicles 22. For example, each autonomous vehicle 22that is not in an error condition could be shown in one color, such asgreen, for example. Any autonomous vehicles 22 that is experiencing anerror condition, such as the specific autonomous vehicle 22′ shown inFIG. 8 , may be shown in a different color, such as yellow or red, forexample, depending on the urgency of the error condition. In someembodiments, the display 58 is configured as a touchscreen display,permitting an operator 8 to interact directly with the display toprovide inputs to the workstation 58. The workstation 58 may be aspecial purpose computer device, or may be a general purpose computerdevice with operating systems, firmware, and software speciallyconfigured to perform the functions described herein.

The remote monitoring station 30 also includes an augmented realityinterface 64 that is worn by an operator 8 when the operator 8 ismonitoring the one or more material storage facilities 10 as shown inFIG. 5 . The augmented reality interface 64 includes a head set 66, and,optionally, may include a pair of gesture sensors 68, 70 worn by theoperator 8 or position tracking sensor 146 that monitors movements andgestures of the operator 8. The head set 66 provides a heads up displayof information presented to a display 72 of the head set 66. Inoperation, the augmented reality interface 64 is not active duringnormal operations of a group of autonomous vehicles 22 being monitoredby an operator 8. The operator 8 is able to see through the display 72and perform normal procedures in the remote monitoring station 30.However, if a particular autonomous vehicle 22 experiences an errorcondition, the operator 8 may select the particular autonomous vehicle22 on the display 58 of the workstation 56 to activate the augmentedreality interface 64 as a virtual controller for the particularautonomous vehicle 22. When the autonomous vehicle 22 is selected, theaugmented reality interface 64 activates the display 72 and places inthe operator 8 in a virtual operator compartment for the particularautonomous vehicle 22, presenting a real-time augmented display usinginformation from the array of cameras 42, accelerometer(s) 44, andcontact sensor array 46, and optionally the LiDAR array 54 and/or sonararray 56 so that the operator 8 may operate the autonomous vehicle 22virtually, from the remote monitoring station 30. As will be explainedbelow, the view presented to the operator 8 will be dependent on theposition and orientation of the operator’s head so that as the operator8 moves her head in real space, the augmented view presented to theoperator 8 on the display 72 changes with the signals from the array ofcameras 42, accelerometer(s) 44, and contact sensor array 46, andoptionally the LiDAR array 54 and/or sonar array 56 being presented in acomposite image that allows the operator 8 to comprehend the currentstate of the environment, including the relative distance of certainelements of the environment surrounding the autonomous vehicle 22. Thegesture sensors 68, 70 allow the operator 8 to activate virtual controlsfor the autonomous vehicle 22, such as a virtual joystick drivercontroller, for example.

Referring now to FIG. 3 , a process 160 for managing data collected bythe sensors 150 of the autonomous vehicle 22 includes a start-up at step162. Once the autonomous vehicle 22 is started at step 162, a start-upprocess step 164 initializes all of the functions and systems of theautonomous vehicle 22 are initialized. The process 160 then advances toa process step 166 where a pre-defined alerts configuration is appliedto the controller 34. The alerts configuration is defined separately ata process 168 and transmitted to the controller 34 by the server 78 ofthe remote monitoring station 30. The pre-defined alerts configurationis established based on the operating environment of the particularautonomous vehicle 22.

Once the alerts configuration is loaded at step 166, the process 160advances to a normal operating process 170. The process continuouslyprogresses through a decision tree where the data from the sensors 150is evaluated to determine if a critical event has occurred at decisionstep 172. If no critical event is detected, the process 160 progressesto a decision step 174 to determine if any important events haveoccurred. If a critical event is detected at decision step 172, then theprocess 160 advances to a step 176 wherein critical event data is savedto the server 78. Then the process 160 progresses to step 178 where theevent data is displayed to the operator 8 at the remote monitoringstation 30.

If no critical event is detected at step 172 and no important event isdetected at step 174, then the process advances to step 180 where sensordata is temporarily saved as short-term data. This short-term data isnot maintained beyond a predetermined period as the data set becomesoverwhelming for the controller 34. However, the short-term event datais available to the operator 8 in certain situations as discussed below.If important data is detected at step 174, then the process 160 advancesto step 178 discussed above and then advances to the step 180.

From step 180, the process advances to step 182 where the autonomousvehicle 182 operates based on the detected data. The operation isevaluated at decision step 184 to determine if operation of theautonomous vehicle 22 should continue. If the data indicates that theautonomous vehicle 22 is safe to operate, then the process 160 returnsto step 170 and progresses through the decision tree described above. Ifthe autonomous vehicle 22 is not safe to operate, then the decision step184 initiates a shut-down and the process progresses to and end step 186where the autonomous vehicle 22 is inoperable until the shut-downcondition is resolved by the operator 8.

Referring to FIG. 14 , a diagrammatic representation of the data flowbetween the autonomous vehicle 22, the server 78, and the workstation 56is shown. The data from the sensors 150 is provided to the controller 34of the autonomous vehicle 22 which shares that data as live event data246 with the remote monitoring station 30. The data from the sensors 150is also applied to an important message filter 230 imbedded in thesoftware of the controller 34 and, when an important event is detected,the important event data 248 is transferred to server storage 236. If acritical event is detected by the critical event filter 232 of thecontroller 34, then the event data 248 is transferred to the serverstorage 236. In addition, upon detection of a critical event by thecritical event filter 232, data stream 250 moves the data from thesensors 150 to a recording storage location 238 in the memory of theserver 78. The event data 248 is applied to emergency filters 234 on theserver 78 to determine if an emergency condition is detected. If so,alerts 242 are forwarded to the remote monitoring station 30. The datastream 250 placed in recording storage 238 is available to the remotemonitoring station 30 through a transfer 244 of recorded event data. Inaddition, the remote monitoring station 30 is operable to update filtersettings 240 which may be applied to the filters 230, 232 of theautonomous vehicle 22 by the server 78 at process step 166 discussedabove.

Thus, when an event occurs requiring the operator 8 to assess theenvironment of the autonomous vehicle 22, the operator 8 may also beable to move back in time, virtually, to view the environment of theautonomous vehicle 22 at the earlier time based on the data saved to theautonomous vehicle at step 180 or the server at step 176. The operator 8may move their head to view different fields of view around theautonomous vehicle 22 at that point in time, with the information on thedisplay being presented as an augmented reality image. This is usefulfor the operator 8 to reconstruct the environment of the autonomousvehicle 22 prior to the error condition. The data from the array ofcameras 42, accelerometer(s) 44, and contact sensor array 46, andoptionally the LiDAR array 54 and/or sonar array 56 is stored in memory38 and is used to reconstruct the image on the augmented realityinterface 64 as if it were in real-time. As such, the operator 8 is ableto detect any abnormalities, improving the ability to resolve the alertcondition.

The workstation 56, through communications circuitry 76, is also incommunication with the cameras 32 positioned throughout the workspace 20and selectively accessible by the operator 8 to choose a particularcamera 32 to view the workspace 20 from the perspective of theparticular camera 32. A server 78 facilitates the communication betweenthe workstation 56, cameras 32, and autonomous vehicles 22 so that allof the information is available to operator 8. This functionality isavailable in real-time such that when an error condition is triggered,such as with the autonomous vehicle 22′ in FIG. 8 , for example, theoperator 8 may choose a particular camera 32, such as camera 32′ in FIG.8 , that has the autonomous vehicle 22′ in its field of view to reviewthe environment of the autonomous vehicle 22′ from the perspective ofthe camera 32. This may occur prior to the operator 8 engaging theaugmented reality interface 64 for the autonomous vehicle 22′. In thisway, the operator 8 will have some context as to the environment of theautonomous vehicle 22′ before becoming the virtual operator 8 of theautonomous vehicle 22′.

The data from the cameras 32 are maintained in memory 74 on theworkstation 56 or at the server 78. When the operator 8 has time-shiftedthe information presented on the augmented reality interface 64 asdiscussed above, the workstation 56 receives the time-shift informationand resets the images from the various cameras 32 to the same point intime as the -time-shifted such that the operator 8 may toggle betweenthe virtual operator condition and viewing the workstation 56 at thetime-shift to thereby review the view from any of the cameras 32 tofurther diagnose the error condition. This further assists the operator8 in diagnosing the error condition of the autonomous vehicle 22′.

Importantly, the workstation 56, through the server 78, is also operableto reset available information in all of the adjacent autonomousvehicles 22 to the time-shifted point in time. Thus, while the operator8 is engaged with a virtual time-shift, any of the actions the operator8 takes will be in reference to the particular point in time that theoperator 8 has shifted to, until the time-shift is released. Forexample, the operator 8 can move from being the virtual operator 8 ofthe autonomous vehicle 22′ if FIG. 8 to the virtual operator 8 of theautonomous vehicle 22″ in FIG. 8 and will have the perspective ofautonomous vehicle 22″ to view the autonomous vehicle 22′ as theoperator 8 diagnoses an error condition. In this way, the availableviews and information for the operator 8 is expanded to include othermembers of the fleet of the autonomous vehicles 22. The time-shift maybe completed by selecting a particular point in real-time, throughtypical A/V playback controls on the augmented reality interface 64 orworkstation 56, or by an error message 126 as shown in FIG. 12 . Becausethe error message is time-stamped, selecting the time-stamp will mayautomatically move the various information stored in memory 38 and/ormemory 74 to that particular point in time to allow the operator 8 toquickly make the virtual time-shift in the augmented reality interface64.

As shown in FIG. 6 , the field of view 80 available on the display 72 islimited. It should be understood that the representations of the fieldof view 80 in FIGS. 6, 7, 9, and 10 that follow are presented asexamples only. The shape and size of the field of view 80 may varydepending on application and the information presented in the augmentedview presented on display 72 may vary depending on the neededinformation. In the view shown in FIG. 6 , an arrow indicator 82provides the operator 8 with a reference of the view being presented tothe front of the particular autonomous vehicle 22. As shown in FIG. 10 ,the orientation of the arrow 82 changes as the operator 8 moves her headand changes the field of view. FIG. 6 shows a typical view of theenvironment around the autonomous vehicle 22. In practice, this displaywill normally be in high definition color. The view of FIG. 6 shows theautonomous vehicle 22″ from the view of autonomous vehicle 22′. Itshould be understood that the view of FIG. 6 omits background that wouldbe visible to improve the clarity of the discussion here.

FIG. 7 shows the view of FIG. 6 , with the display 72 being updated toinclude augmentation from the array of cameras 42, accelerometer(s) 44,and contact sensor array 46, and optionally the LiDAR array 54 and/orsonar array 56. In addition, other feedback may be presented such asground speed 86 from the drive controller 48 or the level of charge 90of the battery 88. The augmentation presented also includes indicationsof the distance of various objects and points as indicated by the dashedlines presented in FIG. 7 . For example, the heavier dashed lines 92 inFIG. 7 indicate that the autonomous vehicle 22″ is relatively close tothe autonomous vehicle 22′. The thinner lines 94 indicate that thosefeatures are farther away. Similarly, the dashed lines 96 indicate thatthe pallet 98 is relatively far away. In the present disclosure, thethickness of the lines 92, 94, and 96 varies to provide informationregarding the relative distance. It should be understood that inimplementation, the lines may be presented differently, includingdifferent colors and may also blink or flash at varying rates to provideappropriate feedback to the operator 8.

Referring to FIG. 12 , an example of a status screen 110 shown on thedisplay 58 of the workstation 56 would be communicated to the operator 8with the autonomous vehicle 22′ being shown in a critical condition. Thedisplay includes a diagrammatic representation 112 of the materialstorage facility 10, which is designated as Plant 1. A status board 114provides information about the various autonomous vehicles 22 indisparate material storage facilities 10, including Plant 1, a Plant 2,and a Plant 3. The first column 116 shows the number of autonomousvehicles 22 operating normally. The second column 118 shows the numberof autonomous vehicles 22 in each material storage facility 10 that hasa cautionary condition. The column 120 shows the number of autonomousvehicles 22 with error conditions. A message screen 122 provides atextual explanation 126 of the error condition, along with a time-stamp124.

FIG. 11 is similar to FIG. 12 , but shows that there are no errorconditions. However, as shown in FIG. 11 , an autonomous vehicle 22W isshown is highlighted to indicate a warning condition. The message screen122 shows a message 130 with a time-stamp 132 that indicates that theautonomous vehicle 22W is having battery performance issues.

FIG. 8 shows that the error condition of FIG. 12 is caused by anobstruction/box 100 that has fallen in the path of the autonomousvehicle 22′. However, because the obstruction 100 is not shown in thefield of view of the augmented display 72 as shown in FIG. 9 , theoperator may move the field of view by moving her head as shown in FIG.10 to see the obstruction 100. At this point, the operator 8 will beable to use the functionality described above to pick different views tosee the obstruction 100 from either camera 32′ or from the perspectiveof autonomous vehicle 22″ and make efforts to resolve the issue.

FIG. 13 illustrates the process 210 of development of the compositeimages seen by the operator 8 when using the augmented reality interface64. The process 210 includes the use of data 200 from the autonomousvehicle 22 or data 202 stored on the server 78 which is chosen at the“or” step 204. Data 206 regarding the performance of the autonomousvehicle 22 is transferred to a data receiver 212 of the server 78. Inaddition, data 208 from various sensors 150 presented in the compositeimage is also provided to the data receiver 212. The three-dimensionaldata is used to generate a point cloud 216 of three-dimensional datawithin the operator’s field of view 218. The point cloud 216 isgenerated to establish an image plane 222 that presents thethree-dimensional data in a two-dimensions at the image plane 222.Additional data is provided through a three-dimensional data streamanalysis 214 of the server 78 so that additional data 220 is presentedat the image plane 222 in composite with the point cloud 216 so that acomposite image including image data and representation of the data fromthe sensors 150 is presented at the image plane 222, providing theoperator 8 the composite image discussed above.

Although this disclosure refers to specific embodiments, it will beunderstood by those skilled in the art that various changes in form anddetail may be made without departing from the subject matter set forthin the accompanying claims.

1. A vehicle tracking system comprising (i) at least one vehicleincluding a control system, the control system including a drive systemproviding power for the vehicle, the drive system operable to move thevehicle and to operate accessories of the vehicle, a plurality ofsensors providing signals indicative of real-time information regardingthe location and operation of the vehicle, a camera system providing asignal representative of images of the environment surrounding thevehicle, a controller including a processor and a memory device, thememory device including instructions that, when executed by theprocessor, cause the processor to receive signals from the sensors andthe camera system, aggregate the signals to create an array of data thatis a synchronized composite of the sensor signals and the camera signalsthat represents a time-sequenced composite image file that superimposesdata derived from the sensor signals onto the images from the camerasystem, (ii) a centralized monitoring station including a computer incommunication with the controller of the at least one vehicle, thecomputer including a processor and a memory device, a user input devicein communication with the computer, a display device, in communicationwith the computer, wherein the memory device of the computer includesinstructions that, when executed by the processor, cause the processorto process inputs from the user input device to communicate with thecontroller of the vehicle to prompt the controller transmit portions ofthe time-sequenced composite image file under the control of the userinput device, the transmitted image file being received by the computerand displayed by the display device, wherein the time-sequencedcomposite image file is transmitted in real time and the portiontransmitted varies based on the user input to generate a field of viewperceptible to a human, wherein the display device comprises a head setand the user input device is coupled to the headset such that movementof the head set changes the portion of the image file transmittedresponsive to movement of the headset to change the field of view beingdisplayed by the head set, further comprising a plurality of vehicles,the computer in communication with the controller of each of thevehicles such that a user may select any of the plurality of vehiclesand view the surroundings of the particular vehicle using the user inputdevice and the head set, wherein the centralized monitoring station isoperable to provide mission tasks to each of the plurality of vehicles,the centralized monitoring system including a monitor that providescurrent status of each of the plurality of vehicles, further comprisinga plurality of cameras positioned in the working environment of theplurality of vehicles, the cameras providing a signal representative ofimages of portions of the working environment of the plurality ofvehicles, the cameras including memory to store an array of data thatrepresents a time-sequenced image file for the field of view of theparticular camera, and wherein the computer is operable to transmit asignal to each of the vehicles and cameras to simultaneously change thepoint of time that the time-sequenced images are presented such that anoperator may toggle between the views of each of the vehicles and eachof the cameras at a coordinated point in time, the portion of the imagefile being transmitted by each vehicle being responsive to the positionof the head set of the user.
 2. The vehicle system of claim 1, whereinthe position of the head set is calibrated from a particular datum inthe environment of the plurality of vehicles.
 3. The vehicle system ofclaim 1, wherein the position of the head set is calibrated from aneutral position relative to the particular camera or vehicle.
 4. Thevehicle system of claim 1, wherein each of the plurality of vehicles isoperable to transmit an alert condition to the centralized monitoringstation, the alert condition prompting the alert to be logged to theparticular real time of the alert, and wherein the computer is operableto mark the point in time such that a user may choose the time of thealert to view the images from the cameras or vehicles.
 5. The vehiclesystem of claim 1, wherein the user input device includes an input forvarying the point in time which corresponds to the portion of the imagefile being transmitted, such that a user may choose to view the portionof the image file as it existed a different time from current real-time.6. The vehicle system of claim 1, wherein the portion of the image filetransmitted is responsive to the position of the head set, such that theuser may vary the field of view at the different time to view thesurroundings of the vehicle at that point in time.
 7. The vehicle systemof claim 1, wherein the time sequence may be paused such that a user maylook around the vehicle at a single point in time by moving the head setto change the field of view.
 8. A vehicle tracking system comprising aplurality of vehicles, each of the plurality of vehicles comprising acontrol system, the control system including a plurality of sensorsproviding signals indicative of real-time information regarding thelocation and operation of the vehicle, and a camera system providing asignal representative of images of the environment surrounding thevehicle, and a controller including a processor and a memory device, thememory device including instructions that, when executed by theprocessor, cause the processor to receive signals from the sensors andthe camera system, aggregate the signals to create an array of data thatis a synchronized composite of the sensor signals and the camera signalsthat represents a time-sequenced composite image file that superimposesdata derived from the sensor signals onto the images from the camerasystem, a plurality of cameras positioned in the working environment ofthe plurality of vehicles, the cameras providing a signal representativeof images of portions of the working environment of the plurality ofvehicles, the cameras including memory to store an array of data thatrepresents a time-sequenced image file for the field of view of theparticular camera, and a centralized monitoring station including a userinput device, an operator head set, and a computer in communication withthe user input device and operator head set, the computer operable totransmit a signal to each of the vehicles and cameras to simultaneouslychange the point of time that the time-sequenced images are presentedsuch that an operator may toggle between the views of each of thevehicles and each of the cameras at a coordinated point in time, theportion of the image file being transmitted by each vehicle beingresponsive to the position of the operator head set.
 9. The vehiclesystem of claim 8, wherein the centralized monitoring station isoperable to provide mission tasks to each of the plurality of vehicles,the centralized monitoring system including a monitor that providescurrent status of each of the plurality of vehicles.
 10. The vehiclesystem of claim 8, wherein the time-sequenced composite image file istransmitted in real time and the portion transmitted varies based on theuser input to generate a field of view perceptible to a human.
 11. Thevehicle system of claim 8, wherein the portion of the image filetransmitted is responsive to the position of the head set, such that theuser may vary the field of view at the different time to view thesurroundings of the vehicle at that point in time.
 12. The vehiclesystem of claim 8, wherein the time sequence may be paused such that auser may look around the vehicle at a single point in time by moving thehead set to change the field of view.
 13. The vehicle system of claim 8,wherein the computer is in communication with the controller of each ofthe vehicles such that a user may select any of the plurality ofvehicles and view the surroundings of the particular vehicle using theuser input device and the head set.
 14. The vehicle system of claim 8,wherein the position of the head set is calibrated from a particulardatum in the environment of the plurality of vehicles.
 15. The vehiclesystem of claim 8, wherein the position of the head set is calibratedfrom a neutral position relative to the particular camera or vehicle.16. The vehicle system of claim 8, wherein each of the plurality ofvehicles is operable to transmit an alert condition to the centralizedmonitoring station, the alert condition prompting the alert to be loggedto the particular real time of the alert, and wherein the computer isoperable to mark the point in time such that a user may choose the timeof the alert to view the images from the cameras or vehicles.
 17. Avehicle management system comprising a plurality of vehicles, eachvehicle including a vehicle camera system, and a controller including aprocessor and a memory device, the memory device including instructionsthat, when executed by the processor, cause the processor to receivesignals from the vehicle camera system, aggregate the signals to createan array of data that is a represents a time-sequenced image file fromthe vehicle camera system, an environmental camera system providingsignal representative of images of the environment surrounding theplurality of vehicles, the environmental cameras including memory tostore an array of data that represents a time-sequenced image file forthe field of view of the particular camera, and a centralized monitoringstation including a computer in communication with the controller eachvehicle, the computer including a processor and a memory device, a userinput device in communication with the computer, a head set, incommunication with the computer, wherein the memory device of thecomputer includes instructions that, when executed by the processor,cause the processor to process inputs from the user input device tocommunicate with the controller of the vehicle to prompt the controllertransmit portions of the time-sequenced image file under the control ofthe user input device, the transmitted image file being received by thecomputer and displayed by the head set, transmit a signal to each of thevehicles and environmental cameras to simultaneously change the point oftime that the time-sequenced images are presented such that an operatormay toggle between the views of each of the vehicles and each of theenvironmental cameras at a coordinated point in time, the portion of theimage file being transmitted by each vehicle being responsive to theposition of the operator head set.
 18. The vehicle system of claim 17,wherein each of the plurality of vehicles is operable to transmit analert condition to the centralized monitoring station, the alertcondition prompting the alert to be logged to the particular real timeof the alert, and wherein the computer is operable to mark the point intime such that a user may choose the time of the alert to view theimages from the cameras or vehicles.
 19. The vehicle system of claim 17,wherein the user input device includes an input for varying the point intime which corresponds to the portion of the image file beingtransmitted, such that a user may choose to view the portion of theimage file as it existed a different time from current real-time. 20.The vehicle system of claim 17, wherein the portion of the image filetransmitted is responsive to the position of the head set, such that theuser may vary the field of view at the different time to view thesurroundings of the vehicle at that point in time.